Methods for reducing row and column patterns in a digital image

ABSTRACT

A method for performing column/row pattern suppression in a digital input image includes creating a smoothed version of the input image by averaging a set of columns/rows neighboring around the column/row being corrected. A difference image is constructed by subtracting the smoothed image from the input image. New column/row intensities are computed from the difference image. An output image is constructed with suppressed column/row patterns by subtracting the new column/row intensities from the input image.

RELATED APPLICATIONS

This application claims priority from U.S. Provisional PatentApplication Ser. No. 61/487,970, filed May 19, 2011.

BACKGROUND

1. Field of the Invention

The present invention relates to processing of digital images. Moreparticularly, the present invention relates to methods for suppressingrow and column patterns in a digital image.

2. The Prior Art

Structured noise in digital images can have the form of row and columnpatterns, typically produced in a digital camera due to the sensordesign (e.g., signal readout architecture) or various other patternsintroduced to the image via image processing.

In digital cameras, column-correlated gain and offset mismatches, due toslight mismatches in column readout circuits, are common. If thesemismatches are large enough, they may show up as high-frequency columnpatterns in the images. The column-correlated mismatches become morepronounced when using a column-parallel readout scheme, using oneanalog-to-digital converter (ADC) per column. While in most cases thecolumn-correlated mismatches can be calibrated out (this type ofmismatch is usually in the form of an offset, and can be calibrated outeither by dedicated on-chip circuitry or by computing column offsetsfrom a dark frame in calibration), in some cases these mismatches arenot easily modeled (such as when a nonlinear ADC transfer function isused, which results in errors that are not purely offset; this kind oferror is signal intensity dependent). In such cases, column offsetcalibration alone will not remove the column-correlated mismatchescompletely, and high-frequency column patterns may still remain visiblein processed images.

In addition to column patterns, row patterns can also create visibleartifacts in digital images. Although the source of row patterns isdifferent from column patterns (row noise is usually a result ofsampling of the power supply noise onto the pixel output), theirvisibility in the images can still be important, especially underlow-light conditions. Similar to column patterns, row patterns usuallycannot be fully removed by standard approaches, resulting in thepresence of high-frequency row patterns in processed images.

In order for column or row patterns to be visible, average intensity ofa given column or row needs to stand out with respect to the neighboringcolumns or rows, respectively. In this proposed method, the intensity ofeach column/row is replaced by the average of its neighbors. Thisresults in the smoothing of the column/row patterns, which makes themless visible in the processed image.

The most common way of suppressing row and column noise in a digitalimage is applying a local smoothing technique. These techniques usuallyoperate directly on the pixel values by defining a moving window andestimating the pixel value in the central window position using thepixel values located inside the window. The supporting window can havean arbitrary shape, ranging from one-dimensional to varioustwo-dimensional windows. Unfortunately, these techniques often produceunsatisfactory results as they cannot fully remove structured noise andalso tend to suppress the desired image features such as edges and imagedetails. Therefore, a different approach is needed.

BRIEF DESCRIPTION

Methods are disclosed that can effectively suppress row and columnpatterns in the image, without blurring image edges and fine details.Moreover, this method is simple, fast, and easy to implement. Advantagesof the methods of the present invention are the simplicity andeffectiveness of the approach in improving image quality by reducing rowand column patterns visibility without compromising image details.

According to a first aspect of the present invention the intensity ofeach column or row is replaced by the average of its neighbors. Thisresults in the smoothing of the column/row patterns, which makes themless visible in the processed image.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

FIG. 1 is a flow diagram showing an illustrative embodiment of a methodfor column noise suppression according to the present invention.

FIG. 2 is a flow diagram showing an illustrative embodiment of a methodfor row noise suppression according to the present invention.

FIG. 3 is a flow diagram showing an illustrative embodiment of a methodfor column or row noise suppression in a color image according to thepresent invention.

FIG. 4 is a flow diagram showing an illustrative embodiment of a methodfor column or row noise suppression wherein the image is divided intospatial regions for processing according to the present invention.

FIG. 5 is a flow diagram showing an illustrative embodiment of a methodfor column or row noise suppression wherein the image is divided intointensity bands for processing according to the present invention.

FIG. 6 is a flow diagram showing an illustrative embodiment of anotherillustrative method for column or row noise suppression according to thepresent invention.

FIG. 7 is a flow diagram showing another illustrative embodiment of amethod for column or row noise suppression according to the presentinvention.

DETAILED DESCRIPTION

Persons of ordinary skill in the art will realize that the followingdescription of the present invention is illustrative only and not in anyway limiting. Other embodiments of the invention will readily suggestthemselves to such skilled persons.

The methods of the present invention can be used to suppress both rowand column noise by applying the smoothing operation in the horizontaland vertical direction, respectively. In this document, the proposedsmoothing concept is described in the context of column noisesuppression. Using the same concept to suppress row noise isstraightforward, that is, applying it to the rows (horizontal direction)instead of columns (vertical direction).

Referring first to FIG. 1 an illustrative method 10 is described. Themethod begins at reference numeral 12. In the case of column noisesuppression, at reference numeral 14, first a smoothed version of theinput image is created by averaging a set of columns neighboring aroundthe column being corrected. According to one embodiment of theinvention, between 1 and 7 columns are used on either side of the columnbeing corrected. Then, at reference numeral 16, a difference image isconstructed by subtracting the smoothed image from the input image.

Since column mismatches may be intensity dependent and tend to be morevisible in flat regions of the image, it would be ideal to average onlythe portions of the scene where there are flat areas. The smoothed imageis going to be blurry compared to the original image. This means thatthe value of the pixels at the edges can change significantly. Incalculating the new column intensities, the pixels (referred to hereinas “edge pixels”) whose values change by more than a pass threshold willnot be taken into account. The value of the pass threshold is determinedempirically, based on subjective observation.

At reference numeral 18, the new column intensities are computed fromthe difference image with the edge pixels taken out. In one exemplaryembodiment of the invention, a correction value is subtracted from theimage.

At reference numeral 20, the image with suppressed column patterns isconstructed by subtracting the new computed column intensities from theinput image. This is performed on a pixel by pixel basis. To prevent thealgorithm from failing in cases where vertically-aligned high-contrastfeatures are present in the image and column averaging will result inpartial transfer of the high-contrast pattern to the adjacent columns, aparameter is introduced which defines the maximum allowable correctionamount. This parameter may be determined empirically (e.g., in thecamera calibration phase) based on the evaluation of test images. Atreference numeral 22 it is determined whether a column average of thedifference image exceeds this parameter value. If the column average ofthe difference image does not exceed this parameter value, an outputimage is constructed at reference numeral 24 and the method ends atreference numeral 26. If the column average of the difference imageexceeds this parameter value, then no correction will be performed onthat particular column as reflected at reference numeral 28. The methodthen ends at reference numeral 26.

In the case of row noise suppression, a method 30 is performed in thehorizontal direction as will now be illustrated with reference to FIG.2. The method begins at reference numeral 32. In the case of row noisesuppression, at reference numeral 34, first a smoothed version of theinput image is created by averaging a set of rows neighboring around therow being corrected. According to one embodiment of the inventionbetween 1 and 7 rows are used on either side of the row being corrected.Then, at reference numeral 36, a difference image is constructed bysubtracting the smoothed image from the input image.

Since row mismatches may be intensity dependent and tend to be morevisible in flat regions of the image, it would be ideal to average onlythe portions of the scene where there are flat areas. The smoothed imageis going to be blurry compared to the original image. This means thatthe value of the pixels at the edges can change significantly. Incalculating the new row intensities, the pixels (referred to herein as“edge pixels”) whose values change by more than a predetermined passthreshold will not be taken into account.

At reference numeral 38, the new row intensities are computed from thedifference image with the edge pixels taken out. In one exemplaryembodiment of the invention, a correction value is subtracted from theimage.

Next, at reference numeral 40, the image with suppressed row patterns isconstructed by subtracting the new computed row intensities from theinput image, done on a pixel by pixel basis. To prevent the algorithmfrom failing in cases where horizontally-aligned high-contrast featuresare present in the image and row averaging will result in partialtransfer of the high-contrast pattern to the adjacent rows, a parameteris introduced which defines the maximum allowable correction amount. Ifa row average of the difference image exceeds this parameter value, thenno correction will be performed on that particular row. At referencenumeral 42 it is determined whether a row average of the differenceimage exceeds this parameter value. If the row average of the differenceimage does not exceed this parameter value, an output image isconstructed at reference numeral 44 and the method ends at referencenumeral 46. If the row average of the difference image exceeds thisparameter value, then no correction will be performed on that particularrow as reflected at reference numeral 48. The method then ends atreference numeral 46.

Should both row and column patterns be suppressed in the image, themethods shown in FIGS. 1 and 2 are both performed on the image. Theproposed algorithm is applied to the input first in the vertical (orhorizontal) direction to produce an intermediate image which should bethen processed by applying the proposed algorithm to the intermediateimage in the horizontal (or vertical) direction. This completes theproposed smoothing procedure.

In one example, the input image is directly passed through the proposedsmoothing procedure. The procedure may include the edge detectionprocess to localize the slowly varying image regions. In the case ofgrayscale image, the application of the proposed solution isstraightforward. In the case of color image, each color channel shouldbe processed separately.

Referring now to FIG. 3, a column or row noise-suppression method 50 fora color image is now shown. The method begins at reference numeral 52.First, at reference numeral 54, a color channel of the image isselected. At reference numeral 56, a smoothed version of the input imageis created by averaging a set of columns (or rows) neighboring aroundthe column (or row) being corrected. Then, at reference numeral 58, adifference image is constructed by subtracting the smoothed image fromthe input image.

As in noise suppression of a grayscale image, it would be ideal toaverage only the portions of the scene where there are flat areas. Thesmoothed image is going to be blurry compared to the original image.This means that the value of the pixels at the edges can changesignificantly. In calculating the new intensities, the pixels (referredto herein as “edge pixels”) whose values change by more than apredetermined pass threshold will not be taken into account.

At reference numeral 60, the new column (or row) intensities arecomputed from the difference image with the edge pixels taken out. Inone exemplary embodiment of the invention, a correction value issubtracted from the image. In other embodiments of the invention, theresult of edge detection applied either to the selected color channel orthe combination (e.g., average or weighted average) of at least twocolor channels, can be used to guide the smoothing process in each colorchannel.

At reference numeral 62, the image with suppressed column (or row)patterns is constructed by subtracting the new computed columnintensities from the input image. This is performed on a pixel by pixelbasis. To prevent the algorithm from failing in cases wherevertically-aligned (or horizontally-aligned) high-contrast features arepresent in the image and averaging will result in partial transfer ofthe high-contrast pattern to the adjacent columns (or rows), a parameteris introduced which defines the maximum allowable correction amount.This parameter may be determined empirically (e.g., in the cameracalibration phase) based on the evaluation of test images. At referencenumeral 64 it is determined whether a column (or row) average of thedifference image exceeds this parameter value. If the column (or row)average of the difference image does not exceed this parameter value,correction to the color channel is applied at reference numeral 66. Ifthe column (or row) average of the difference exceeds this parametervalue, then no correction will be performed on that particular column(or row) as reflected at reference numeral 68.

At reference numeral 70, it is determined whether all color channelshave been processed. If not, the method returns to reference numeral 54,where another color channel is selected for processing. If all colorchannels have been processed, the method proceeds to reference numeral72, where an output image is constructed. The method then ends atreference numeral 74.

In another example, the input image is divided into a number of eithernon-overlapping or overlapping spatial regions. The proposed smoothingprocedure, that may include the edge detection process, is applied toeach of these spatial regions to produce their enhanced versions. Thefinal image is obtained by spatially combining the enhanced regions.This embodiment of the invention is illustrated in FIG. 4.

Referring now to FIG. 4, an illustrative method 80 is described. Themethod begins at reference numeral 82. At reference numeral 84, theimage is divided into spatial regions. Division of the image may beaccomplished by means of image segmentation or by simply splitting theimage into regular-sized square or rectangular blocks. Next, atreference numeral 86, one of the spatial regions is selected forprocessing.

At reference numeral 88, a smoothed version of the input image iscreated by averaging a set of columns or rows neighboring around thecolumn or row being corrected. According to one embodiment of theinvention between 1 and 7 columns are used on either side of the columnbeing corrected. Then, at reference numeral 90, a difference image isconstructed by subtracting the smoothed image from the input image. Aswith the other embodiments of the present invention, the pixels(referred to herein as “edge pixels”) whose values change by more than apredetermined pass threshold will not be taken into account in thisprocess.

At reference numeral 92, the new column or row intensities are computedfrom the difference image with the edge pixels taken out. In oneexemplary embodiment of the invention, a correction value is subtractedfrom the image.

At reference numeral 94, the image with suppressed column patterns isconstructed by subtracting the new computed column intensities from theinput image. This is performed on a pixel by pixel basis. To prevent thealgorithm from failing in cases where vertically-aligned orhorizontally-aligned high-contrast features are present in the image andcolumn or row averaging will result in partial transfer of thehigh-contrast pattern to the adjacent columns or rows, a parameter isintroduced which defines the maximum allowable correction amount. Thisparameter may be determined empirically (e.g., in the camera calibrationphase) based on the evaluation of test images. At reference numeral 96it is determined whether an average of the difference image exceeds thisparameter value. If the average of the difference does not exceed thisparameter value, an output spatial region is constructed at referencenumeral 98. If the average of the difference exceeds this parametervalue, then no correction will be performed on that particular column orrow as reflected at reference numeral 100.

At reference numeral 102, it is determined whether all spatial regionshave been processed. If not, then process returns to reference numeral86, where another spatial region is selected for processing. If allspatial regions have been processed, the new values for all spatialregions are combined to construct an output image at reference numeral104. The method then ends at reference numeral 106.

In another example, the input image is divided into eithernon-overlapping or overlapping intensity bands. The proposed smoothingprocedure, that may include the edge detection process, is applied toeach of these intensity bands to produce their enhanced versions. Thefinal image is obtained by combining the enhanced intensity bands. Anexemplary embodiment of this process is illustrated in FIG. 5.

Referring now to FIG. 5, an illustrative method 110 is described. Themethod begins at reference numeral 112. At reference numeral 114, theimage is divided into intensity bands. Division of the image may beaccomplished by means of multi-level thresholding, bit-leveldecomposition, or range decomposition. Next, at reference numeral 116,one of the intensity bands is selected for processing.

At reference numeral 118, a smoothed version of the input image iscreated by averaging a set of columns or rows neighboring around thecolumn or row being corrected. Then, at reference numeral 120, adifference image is constructed by subtracting the smoothed image fromthe input image. As with the other embodiments of the present invention,the pixels (referred to herein as “edge pixels”) whose values change bymore than a predetermined pass threshold will not be taken into accountin this process.

At reference numeral 122, the new column or row intensities are computedfrom the difference image with the edge pixels taken out. In oneexemplary embodiment of the invention, a correction value is subtractedfrom the image.

At reference numeral 124, the image with suppressed column patterns isconstructed by subtracting the new computed column intensities from theinput image. This is performed on a pixel by pixel basis. To prevent thealgorithm from failing in cases where vertically-aligned orhorizontally-aligned high-contrast features are present in the image andcolumn or row averaging will result in partial transfer of thehigh-contrast pattern to the adjacent columns or rows, a parameter isintroduced which defines the maximum allowable correction amount. Thisparameter may be determined empirically (e.g., in the camera calibrationphase) based on the evaluation of test images. At reference numeral 126it is determined whether an average of the difference image exceeds thisparameter value. If the average of the difference does not exceed thisparameter value, an output intensity band is constructed at referencenumeral 128. If the average of the difference exceeds this parametervalue, then no correction will be performed on that particular column orrow as reflected at reference numeral 130.

At reference numeral 132, it is determined whether all intensity bandshave been processed. If not, then process returns to reference numeral86, where another intensity band is selected for processing. If allintensity bands have been processed, the new values for all intensitybands are combined to construct an output image at reference numeral134. The method then ends at reference numeral 136.

In yet another example, illustrated with reference to FIG. 6, a method140 for column or row noise suppression is disclosed. The process beginsat reference numeral 142. At reference numeral 144, the input image isfirst passed through low-pass filtering to create a smoothed version ofthe original image. Then, at reference numeral 146, a so-called residualimage is created as a function (i.e., difference or ratio) of theoriginal image and its smoothed version. Then, at reference numeral 148,the column or row noise suppression procedure disclosed herein isapplied to the residual image to produce an output residual image. Atreference numeral 150, the output residual image is combined (i.e.,addition or multiplication; or generally speaking, using an inversefunction to the function used to generate the residual signal) with thesmoothed version of the original image to produce the output image withsuppressed row and column patterns. The process ends at referencenumeral 152.

In yet another example, illustrated with reference to FIG. 7, analternate method 160 for column or row noise suppression in a colorimage is disclosed. The process begins at reference numeral 162. Atreference numeral 164, the input image is first passed through low-passfiltering to create a smoothed version of the original image. Then, atreference numeral 166, a so-called grayscale residual image is createdas a function (i.e., difference or ratio) of the original image and itssmoothed version. In one example, a grayscale residual image is createdas a function of grayscale versions of the original image and itssmoothed version. In another example, a grayscale version of theresidual image is created by combining the color components of theresidual image in each pixel location using standard averaging orweighted averaging techniques. At reference numeral 168, the column orrow noise suppression procedure disclosed herein is applied to thegrayscale residual image to produce an output residual image. Atreference numeral 170, the final residual image is then combined witheach color channel to produce the output image with suppressed row andcolumn patterns. Combining is performed by adding the final residualimage to each color channel of the smoothed version of the originalimage where the original residual image is obtained as the differencebetween the original image and its smoothed version. Alternatively,combining is performed by multiplying the final residual image with toeach channel of the smoothed version of the original image when theoriginal residual image is obtained as the ratio between the originalimage and its smoothed version. The process ends at reference numeral172.

According to another aspect of the present invention, the variousmethods disclosed herein may be combined. For example, the output imagegenerated using one of the disclosed methods can be used as the inputfor some other of the disclosed methods to further improve the smoothingperformance. Alternatively, the output images generated using at leasttwo different methods can be blended or combined. Eventually, any of thedisclosed methods can be applied in more than one iteration.

While embodiments and applications of this invention have been shown anddescribed, it would be apparent to those skilled in the art that manymore modifications than mentioned above are possible without departingfrom the inventive concepts herein. The invention, therefore, is not tobe restricted except in the spirit of the appended claims.

What is claimed is:
 1. A method for performing column/row patternsuppression in a digital input image, comprising: creating a smoothedversion of the input image by averaging a set of columns/rowsneighboring around the column/row being corrected; constructing adifference image by subtracting the smoothed image from the input image;computing new column/row intensities from the difference residual image;constructing an output image with suppressed column/row patterns bysubtracting the new column/row intensities from the input image,including defining a maximum allowable correction amount and performingno correction on a column/row if the column/row average of thedifference image exceeds the maximum allowable correction amount.
 2. Themethod of claim 1 wherein computing new column/row intensities from thedifference image comprises ignoring pixels whose values change by morethan a pass threshold.
 3. The method of claim 1 wherein the image is acolor image and wherein each color channel of the image is processedindividually and combined.
 4. A method for performing column/row noisesuppression in a digital input image, comprising: creating a smoothedversion of the original image by passing the input image throughlow-pass filtering to create a smoothed image; creating a residual imageas a function of the original image and the smoothed image; creating asmoothed version of the residual image by averaging a set ofcolumns/rows neighboring around the column/row being corrected;constructing a difference residual image by subtracting the smoothedresidual image from the original residual image; computing newcolumn/row intensities from the difference residual image; constructinga final residual image with suppressed column/row patterns bysubtracting the new column/row intensities from the original residualimage, including defining a maximum allowable correction amount and,performing no correction on a column/row if the column/row average ofthe difference residual image exceeds the maximum allowable correctionamount; producing an output image by combining the final residual imagewith the smoothed image to produce the output image.
 5. The method ofclaim 4 wherein creating a residual image as a function of the originalimage and the smoothed image comprises subtracting the smoothed imagefrom the original image.
 6. The method of claim 4 wherein creating aresidual image as a function of the original image and the smoothedimage comprises generating a ratio of the smoothed image and theoriginal image.
 7. The method of claim 4 wherein computing newcolumn/row intensities from the difference residual image comprisesignoring pixels whose values change by more than a pass threshold. 8.The method of claim 4 wherein producing an output image by combining thefinal residual image with the smoothed image to produce the output imagecomprises adding the final residual image to the smoothed image toproduce the output image.
 9. The method of claim 4 wherein producing anoutput image by combining the residual image with the smoothed image toproduce the output image comprises multiplying the final residual imagewith the smoothed image to produce the output image.